Skip to content

fix: update kernel pin and keep the close-race error typed - #619

Merged
vuanhphung merged 5 commits into
mainfrom
vuanhphung/kernel-pin-release-session-at-close
Nov 17, 2026
Merged

vuanhphung merged 5 commits into
mainfrom
vuanhphung/kernel-pin-release-session-at-close

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Updates KERNEL_REV from 0b5d0ed to dac32e2, the current kernel main. The pin picks up databricks/databricks-sql-kernel#462: AsyncStatement.close() now releases its canceller and the result streams of the AsyncResultHandles it returned. Before, a closed async statement kept the kernel session, and the host's shared telemetry client, alive until JS garbage collection. A later session on the same host then reused that client and ignored its own telemetryBatchSize / telemetryFlushIntervalMs. This fixes TELEMETRY-015 on SEA, which databricks/databricks-driver-test#2069 tracks as an xfail tripwire.

The pin also picks up databricks/databricks-sql-kernel#451 (reuse a successful async terminal response instead of re-polling), #473 (OSV dependency fixes) and #454 (drop presigned query strings from request errors and debug URLs). The old pin 0b5d0ed was the #436 branch head; its tree matches #436's merge on main (db23442). The regenerated native/kernel/index.d.ts only adds #462's doc comments on AsyncStatement.close() and AsyncResultHandle.fetchNextBatch().

Close during an in-flight fetch. Because #462 empties the result handle on close, an SEA operation closed while a multi-batch fetch is in flight made that fetch reject with the kernel's InvalidStatementHandle, surfaced as a generic HiveDriverError: AsyncResultHandle already closed. On the old pin it rejected with OperationStateError(CLOSED). fetchFailure(), which handles a failed fetchChunk() or hasMore(), now records whether a local close or cancel happened, runs its usual cleanup close, and then reports OperationStateError(Closed/Canceled) instead of the decoded kernel error. Keeping the cleanup close matters after a cancel: nothing else closes the statement then. The fetch still stops early, as #462 intends. #462 also empties the handle when the close RPC fails, while kernelClose rolls isClosed back so the close can be retried. So close() now sets a closeRequested flag that a failure doesn't clear (cleanup closes don't set it), and fetchFailure() treats it as a local close. A fetch hit by a failed close now gets OperationStateError(CLOSED) instead of the HiveDriverError; on the old pin it kept returning rows.

Validation, with the native binding built from the new pin:

  • TypeScript build, lint, prettier --check, type-check, and 1,785 unit tests. The new KernelOperationBackend — close()/cancel() during an in-flight fetch tests (fetchChunk() and hasMore()) assert the typed error and a single statement.close(), and cover a close() whose RPC fails (retry resolving, as in the kernel, or rejecting), which takes one cleanup close. Two concurrent fetches that fail on the same error both report that error, not CLOSED. Without the change the close and failed-close cases fail with the HiveDriverError.
  • Failed close on the binding itself (kernel wiremock probe, DELETE answered with 400): on 0b5d0ed the next fetchNextBatch() still returns a batch; on dac32e2 it rejects with InvalidStatementHandle. A second close() resolves on both.
  • Interrupt-race probe on SEA (runAsync SELECT id, uuid() FROM range(0, 3000000), fetch 1,000 rows, start fetchChunk({ maxRows: 3e6 }), interrupt after 1 s):
    • close(): OperationStateError CLOSED in 3/3 runs, settling in about 1.07 s (4/4 again with the failed-close change, about 1.09 s). Without the change, 3/3 runs give HiveDriverError "AsyncResultHandle already closed".
    • cancel() with CloudFetch off (paged results, so the next page fails): OperationStateError CANCELED in 3/3 runs (4/4 again with the failed-close change), and the kernel statement is closed.
  • Live Node SEA against the pecotesting warehouse, scoped driver-test runs on main (e40743249):
    • Non-blocking statement execution, runAsync mode, Close, Cancel, Use after close, State machine, fetchAll, fetchChunk, Status polling, Close cascade, and Statement close invalidates held op: 39 passing, 0 failing, with and without the change.
    • The telemetry file (TELEMETRY-0*), serial, under --max-semi-space-size=256: 14 passing, including TELEMETRY-015.
    • ExecuteStatement (basic), ResultFetching, Structured error on bad SQL, and Client.openSession(): 21 passing.
  • The TELEMETRY-015 SEA tripwire in databricks/databricks-driver-test#2069 (2789079b0) throws XPASS in 2/2 runs. That PR flips the cell to implemented before it merges.

Merge prerequisite: databricks/databricks-driver-test#2146 re-records seven Node SEA replay recordings for kernel#451's dropped re-poll, and must merge before this PR enters the merge queue.

This pull request and its description were written by Isaac.

Pin the kernel to dac32e2 on main. It includes
databricks/databricks-sql-kernel#462: AsyncStatement.close() now releases
its canceller and the result streams of its handles, so a closed async
statement no longer keeps the session, and the host's shared telemetry
client, alive until GC. It also picks up #451, #473 and #454.

The regenerated native/kernel/index.d.ts only adds #462's doc comments.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
…rror

With the new kernel pin, AsyncStatement.close() empties the result handle,
so a fetchChunk() or hasMore() in flight on SEA rejects with the kernel's
InvalidStatementHandle ("AsyncResultHandle already closed"), surfaced as a
generic HiveDriverError. Check the lifecycle first in their catch blocks so
a local close or cancel still reports OperationStateError(Closed/Canceled).

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung vuanhphung changed the title fix: update kernel pin to release closed async statements fix: update kernel pin and keep the close-race error typed Nov 16, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly good: the pin bump and the typed close-race error are sound. One medium regression: the new guard also catches local cancels, so a cancel during an in-flight fetch no longer closes the kernel statement, and the facade never closes it after a cancel.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
The guard in the fetchChunk() and hasMore() catch blocks threw before
fetchFailure() ran, so a fetch interrupted by cancel() skipped the cleanup
close. After a cancel, nothing else closes the statement, so it and its
session stayed alive until GC.

Move the check into fetchFailure(): record whether a local close or
cancel happened, run the cleanup close, then report the lifecycle state.
The flags are read first because the cleanup close sets isClosed.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

With the new kernel pin, AsyncStatement.close() empties the result handle
even when its close RPC fails, but kernelClose rolls isClosed back. A
fetch that then hits the emptied handle reported the kernel's
HiveDriverError "AsyncResultHandle already closed" instead of
OperationStateError(Closed). On the old pin it kept returning rows.

Record the close() request in a flag that the failure doesn't reset and
treat it in fetchFailure() like a local close. Cleanup closes don't set
it. fetchFailure() now builds the Canceled/Closed error itself, so the
result doesn't depend on its own cleanup close succeeding.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low concern: a concurrent fetch can report OperationStateError(Closed) instead of the real error, because the cleanup close sets isClosed too. The closeRequested flag, the cleanup-close ordering and the new unit tests cover the close, cancel and failed-close races as the PR describes.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
fetchFailure() counted isClosed as a local close, but its own cleanup
close sets that flag before its first await. When two fetches failed on
the same error, the second reported OperationStateError(Closed) instead
of the error. closeRequested now covers every client close(), so check
only it and isCancelled.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 17, 2026

This branch was successfully deployed

1 active deployment
azure-prod — 87b7dfbe Deployed Nov 17, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2085
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant